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Abstract: If the log likelihood is approximately quadratic with constant Hes- 
sian, then the maximum likelihood estimator (MLE) is approximately normally 
distributed. No other assumptions are required. Wc do not need independent 
and identically distributed data. We do not need the law of large numbers 
(LLN) or the central limit theorem (CLT). We do not need sample size going 
to infinity or anything going to infinity. 

Presented here is a combination of Le Cam style theory involving local 
asymptotic normality (LAN) and local asymptotic mixed normality (LAMN) 
and Cramer style theory involving derivatives and Fisher information. The 
main tool is convergence in law of the log likelihood function and its derivatives 
considered as random elements of a Polish space of continuous functions with 
the metric of uniform convergence on compact sets. We obtain results for both 
one-step-Newton estimators and Newton-iterated-to-convergence estimators. 



1. Disclaimer 

Few papers need a disclaimer, but this paper seems to offend some people. For 
the record, your humble author has written papers using conventional asymptotics 
before this paper and will continue to write such papers in the future. Many readers 
find this paper presents an interesting new (to them) way to look at maximum 
likelihood, but some readers get upset about what they take to be its disparagmcnt 
of conventional asymptotics. But this "disparagement" says nothing that every 
statistician does not already know. In any actual data analysis, the sample size 
is not changing. Whether or not more data of the same type may be collected in 
the future has no bearing on the validity of the analysis being done now. Whether 
asymptotics "works" or not for any actual data, cannot be established by checking 
conditions of theorems, since the theorems only say asymptotics "works" for some 
sufficiently large n not necessarily at the n for the actual data. The only practical 
way to check whether asymptotics "works" is to do simulations. 



2. Introduction 

The asymptotics of maximum likelihood is beautiful theory. If you can calculate 
two derivatives of the log likelihood, you can find the maximum likelihood estimate 
(MLE) and its asymptotic normal distribution. 

But why does this work? And when does it work? The literature contains many 
different treatments, many theorems with long lists of assumptions, each slightly 
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different from the others, and long messy calculations in their proofs. But they give 
no insight because neither assumptions nor calculations are sharp. The beauty of 
the theory is hidden by the mess. 

In this article we explore an elegant theory of the asymptotics of likelihood 
inference inspired by the work of Lucien Le Cam, presented in full generality in 
Le Cam (1986) and in somewhat simplified form in Lc Cam and Yang (2000). Most 
statisticians find even the simplified version abstract and difficult. Here we attempt 
to bring this theory down to a level most statisticians can understand. We take 
from Le Cam two simple ideas. 

• If the log likelihood is approximately quadratic with constant Hessian, then 
the MLE is approximately normal. This is the theory of locally asymptotically 
normal (LAN) and locally asymptotically mixed normal (LAMN) models. 

• Asymptotic theory does not need n going to infinity. We can dispense with 
sequences of models, and instead compare the actual model for the actual 
data to an LAN or LAMN model. 

Although these ideas are not new, being "well known" to a handful of theoreti- 
cians, they are not widely understood. When I tell typical statisticians that the 
asymptotics of maximum likelihood have nothing to do with the law of large num- 
bers (LLN) or the central limit theorem (CLT) but rather with how close the log 
likelihood is to quadratic, I usually get blank stares. That's not what they learned 
in their theory class. Worse, many statisticians have come to the conclusion that 
since the "n goes to infinity" story makes no sense, asymptotics are bunk. It is a 
shame that so many statisticians are so confused about a crucial aspect of statistics. 

2.1. Local Asymptotic Normality 

Chapter 6 of Le Cam and Yang (2000) presents the theory of local asymptotically 
normal (LAN) and locally asymptotically mixed normal (LAMN) sequences of mod- 
els. This theory has no assumption of independent and identically distributed data 
(nor even stationarity and weak dependence). The n indexing an LAN or LAMN 
sequence of models is just an index that need not have anything to do with sample 
size or anything analogous to sample size. Thus the LLN and the CLT do not apply, 
and asymptotic normality must arise from some other source. 

Surprising to those who haven't seen it before, asymptotic normality arises from 
the log likelihood being asymptotically quadratic. A statistical model with exactly 
quadratic log likelihood 

(1) 1{6) = U + Z'6-\6'K6, 6eM. p 

where U is a random scalar, Z is a random p vector, and K is a random p x p 
symmetric matrix, has observed Fisher information K and maximum likelihood 
estimator (MLE) 

(2) § = K~ 1 Z 

(if K is positive definite). If K is constant, then the distribution of the MLE is 

(3) 6 ~ JV(8, A' -1 ), 

the right hand side being the multivariate normal distribution with mean vector 6 
and variance matrix K~ x (Corollary 3.2 below). If K is random, but invariant in 
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law, meaning its distribution does not depend on the parameter 9, then (3) holds 
conditional on K (Theorem 3.1 below). 

The essence of likelihood asymptotics consists of the idea that if the log likelihood 
is only approximately of the form (1) with K invariant in law, then (3) holds 
approximately too. All likelihood asymptotics that produce conclusions resembling 
(3) are formalizations of this idea. The idea may be lost in messy proofs, but it's 
what really makes likelihood inference work the way it does. 

2.2. "No n" Asymptotics 

By "no n asymptotics" we mean asymptotics done without reference to any se- 
quence of statistical models. There is no n going to infinity. This is not a new idea, 
as the following quotation from Le Cam (1986, p. xiv) shows. 

Prom time to time results are stated as limit theorems obtainable as something called n 
"tends to infinity." This is especially so in Chapter 7 [Some Limit Theorems] where the results 
are just limit theorems. Otherwise we have made a special effort to state the results in such 
a way that they could eventually be transformed into approximation results. Indeed, limit 
theorems "as n tends to infinity" are logically devoid of content about what happens at any 
particular n. All they can do is suggest certain approaches whose performance must then be 
checked on the case at hand. Unfortunately the approximation bounds we could get were too 
often too crude and cumbersome to be of any practical use. Thus we have let n tend to infinity, 
but we urge the reader to think of the material in approximation terms, especially in subjects 
such as ones described in Chapter 11 [Asymptotic Normality — Global]. 

Le Cam's point that asymptotics "arc logically devoid of content about what 
happens at any particular n" refers to the fact that convergence of a sequence tells 
us nothing about any initial segment of the sequence. An estimator 9 n that is equal 
to 42 for all n < 10 10 and equal to the MLE thereafter is asymptotically equivalent 
to the MLE. Strictly speaking, asymptotics — at least the usual story about n going 
to infinity — does not distinguish between these estimators and says you might just 
as well use one as the other. 

The story about n going to infinity is even less plausible in spatial statistics 
and statistical genetics where every component of the data may be correlated with 
every other component. Suppose we have data on school districts of Minnesota. How 
does Minnesota go to infinity? By invasion of surrounding states and provinces of 
Canada, not to mention Lake Superior, and eventually by rocket ships to outer 
space? How silly does the n goes to infinity story have to be before it provokes 
laughter instead of reverence? 

Having once seen the absurdity of the n goes to infinity story in any context, it 
becomes hard to maintain its illusion of appropriateness in any other context. A 
convert to the "no n" view always thinks n = 1. You always have one "data set," 
which comprises the data for an analysis. And it isn't going to infinity or anywhere 
else. 

But how do "no n asymptotics" work? It is not clear (to me) what Le Cam meant 
by "eventually transformed into approximation results" because he used a conver- 
gence structure (Le Cam and Yang, 2000, Chapter 6, Definitions 1, 2, and 3) so 
weak that, although topological (Beattie and Butzmann, 2002, Proposition 1.7.15), 
it seems not metrizable. So at this point I part company with Le Cam and present 
my own take on "no n asymptotics" (although our Section 3 follows Le Cam closely, 
our Section 4 and Appendix B seem new). It has the following simple logic. 

• All assumptions are packed into a single convergence in law statement involv- 
ing the log likelihood. 
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• The conclusion is a convergence in law statement about an estimator. 

• Hence delta and epsilon arguments using metrics for convergence in law can 
replace sequential arguments. 

One source of this scheme is Geyer (1994), which did not use the whole scheme, 
but which in hindsight should have. The conclusion of Lemma 4.1 in that article is 
a "single convergence in law statement about the log likelihood" that incorporates 
most of the assumptions in that article. It is a forerunner of our treatment here. 

At this point wc introduce a caveat. By "no n" asymptotics we do not mean the 
letter n is not allowed to appear anywhere, nor do we mean that sequences are not 
allowed to be used. We mean that we do not use sequences associated with a story 
about the index n being sample size or anything analogous to sample size. 

We do use sequences as a purely technical tool in discussing continuity. To prove a 
function f : U —> V between metric spaces with metrics du and dy continuous, one 
can show that for every x € U and every e > there exists a S, which may depend 
on x and e, such that dy(f(x), f(y)) < e, whenever du(x, y) < S. Alternatively, one 
can show that f(x n ) — > f(x), whenever The mathematical content is the 

same either way. 

In particular, it is more convenient to use sequences in discussing convergence of 
probability measures on Polish spaces. The space of all probability measures on a 
Polish space is itself a metric space (Billingsley, 1999, p. 72). Hence delta-epsilon 
arguments can be used to discuss convergence, but nearly the whole probability 
literature on convergence of probability measures uses sequences rather than delta- 
epsilon arguments, so it is easier to cite the literature if we use sequences. The 
mathematical content is the same either way. 

The point is that we do not reify n as sample size. So perhaps a longer name 
such as "n not reified as sample size" asymptotics would be better than the shorter 
"no n" asymptotics, but we will continue with the shorter name. 

What is important is "quadraticity." If the log likelihood for the actual model 
for the actual data is nearly quadratic, then the MLE has the familiar properties 
discussed in Section 2.1, but no story about sample size going to infinity will make 
the actual log likelihood more quadratic than it actually is or the actual MLE more 
nearly normal. 

I concede that metrics for convergence in law are unwieldy and might also give 
"approximation bounds . . . too crude and cumbersome to be of any practical use." 
But, unlike Le Cam, I am not bothered by this because of my familiarity with 
computer intensive statistical methods. 

2.3. Asymptotics is Only a Heuristic 

(This slogan means what Le Cam meant by "all they can do is suggest certain 
approaches" .) We know that asymptotics often works well in practical problems be- 
cause we can check the asymptotics by computer simulation (perhaps what Le Cam 
meant by "checked on the case at hand"), but conventional theory doesn't tell us 
why asymptotics works when it does. It only tells us that asymptotics works for 
sufficiently large n, perhaps astronomically larger than the actual n of the actual 
data. So that leaves a theoretical puzzle. 

• Asymptotics often works. 

• But it doesn't work for the reasons given in proofs. 

• It works for reasons too complicated for theory to handle. 
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I am not sure about "too complicated ... to handle." Perhaps a computer-assisted 
proof could give "approximation bounds ... of practical use," what Le Cam wanted 
but could not devise. But when I think how much computer time a computer- 
assisted proof might take and consider alternative ways to spend the computer 
time, I do not see how approximation bounds could be as useful as a parametric 
bootstrap, much less a double parametric bootstrap (since we are interested in 
likelihood theory for parametric models we consider only the parametric bootstrap). 

A good approximation bound, even if such could be found, would only indicate 
whether the asymptotics work or don't work, but a bootstrap of approximately piv- 
otal quantities derived from the asymptotics not only diagnoses any failure of the 
asymptotics but also provides a correction, so the bootstrap may work when asymp- 
totics fails. And the double bootstrap diagnoses any failure of the single bootstrap 
and provides further correction, so the double bootstrap may work when the sin- 
gle bootstrap fails (Bcran, 1987, 1988; Geyer, 1991; Hall, 1992; Newton and Geyer, 
1994). 

With ubiquitous fast computing, there is no excuse for not using the bootstrap 
to improve the accuracy of asymptotics in every serious application. Thus we arrive 
at the following attitude about asymptotics 

• Asymptotics is only a heuristic. It provides no guarantees. 

• If worried about the asymptotics, bootstrap! 

• If worried about the bootstrap, iterate the bootstrap! 

However, the only justification of the bootstrap is asymptotic. So this leaves us 
in a quandary of circularity. 

• The bootstrap is only a heuristic. It provides no guarantees. 

• All justification for the bootstrap is asymptotic! 

• In order for the bootstrap to work well, one must bootstrap approximately 
asymptotically pivotal quantities! 

(the "approximately" recognizes that something less than perfectly pivotal, for 
example merely variance stabilized, is still worth using). 

In practice, this "circularity" does not hamper analysis. In order to devise good 
estimates one uses the asymptotics heuristic (choosing the MLE perhaps). In or- 
der to devise "approximately asymptotically pivotal quantities" one again uses the 
asymptotics heuristic (choosing log likelihood ratios perhaps). But when one cal- 
culates probabilities for tests and confidence intervals by simulation, the calcula- 
tion can be made arbitrarily accurate for any given 9. Thus the traditional role of 
asymptotics, approximating Pg, is not needed when we bootstrap. We only need 
asymptotics to deal with the dependence of Pg on 9. Generally, this dependence 
never goes entirely away, no matter how many times the bootstrap is iterated, but 
it does decrease (Beran, 1987, 1988; Hall, 1992). 

The parametric bootstrap simulates multiple data sets y*, y* from Pg^ y y 

where y is the real data and 9 some estimator. The double parametric bootstrap 
simulates multiple data sets from each P§( y *y Its work load can be reduced by using 
importance sampling (Newton and Geyer, 1994). Assuming 9 i-> Pg is continuous, 
these simulations tell everything about the model for 9 in the region filled out by 
the 9(y*). But nothing prevents one from simulating from 9 in a bigger region if 
one wants to. 

The parametric bootstrap is very different from the nonparametric bootstrap in 
this respect. The nonparametric bootstrap is inherently an asymptotic (large n) 

imsart-coll ver. 2011/11/15 file: Geyer.tex date: March 8, 2013 



6 



C. J. Geyer 



methodology because resampling the data, which in effect substitutes the empirical 
distribution of the data for the true unknown distribution, only "works" when the 
empirical distribution is close to the true distribution, which is when the sample size 
is very large. When the parametric bootstrap simulates a distribution Pg it does so 
with error that does not depend on sample size but only on how long we run the 
computer. When a double parametric bootstrap simulates Pg for many different 9 
values, we learn about Pg for these 9 values and nearby 9 values. If these 9 values 
are densely spread over a region, we learn about Pg for all 9 in the region. So long 
as the true unknown 9 value lies in that region, we approximate the true unknown 
distribution with accuracy that does not depend on sample size. Thus if one does 
enough simulation, the parametric bootstrap can be made to work for small sample 
sizes in a way the nonparametric bootstrap cannot. Since this is not a paper about 
the bootstrap, we will say no more on the subject. 

Nevertheless, asymptotics often does "work" and permits simpler calculations 
while providing more insight. In such cases, the bootstrap when used as a diagnostic 
(Geyer, 1991) proves its own pointlcssncss. A single bootstrap often shows that it 
cannot improve the answer provided by asymptotics, and a double bootstrap often 
shows that it cannot improve the answer provided by the single bootstrap. 

Since asymptotics is "only a heuristic," the only interesting question is what 
form of asymptotics provides the most useful heuristic and does so in the simplest 
fashion. This article is my attempt at an answer. 

2-4- An Example from Spatial Statistics 

Geyer and M0ller (1994) give a method of simulating spatial point processes and 
doing maximum likelihood estimation. In their example, a Strauss process (Strauss, 
1975), they noted that the asymptotic distribution of the MLE appeared to be very 
close to normal, although the best asymptotic results they were able to find in 
the literature (Jensen, 1991, 1993) only applied to Strauss processes with very 
weak dependence, hence very close to a Poisson process (Geyer and M0ller, 1994, 
Discussion), which unfortunately did not include their example. 

From a "no n" point of view, this example is trivial. A Strauss process is a 
two-parameter full exponential family. In the canonical parameterization, which 
Geyer and M0ller (1994) were using, the random part of the log likelihood is linear 
in the parameter, hence the Hessian is nonrandom. Hence, according to the theory 
developed here, the MLE will be approximately normal so long as the Hessian is 
approximately constant over the region of parameter values containing most of the 
sampling distribution of the MLE. 

Geyer and M0llcr (1994) did not then understand the "no n" view and made no 
attempt at such verification (although it would have been easy). Direct verification 
of quadraticity is actually unnecessary here, because the curved part of the log 
likelihood (in the canonical parameterization) of an exponential family is propor- 
tional to the cumulant generating function of the canonical statistic, so the family 
is nearly LAN precisely when the distribution of the canonical statistic is nearly 
normal, which Geyer and M0ller (1994) did investigate (their Figure 2). 

Thus there is no need for any discussion of anything going to infinity. The asymp- 
totics here, properly understood, are quite simple. As we used to say back in the 
sixties, the "n goes to infinity" story is part of the problem not part of the solution. 

Although the exponential family aspect makes things especially simple, the same 
sort of thing is true in general. When the Hessian is random, it is enough that it 
be nearly invariant in law. 

imsart-coll ver. 2011/11/15 file: Geyer.tex date: March 8, 2013 



Maximum Likelihood 



7 



2.5. An Example from Statistical Genetics 

Fisher (1918) proposed a model for quantitative genetics that has been widely used 
in animal breeding and other areas of biology and indirectly led to modern regression 
and analysis of variance. The data Y are multivariate normal, measurements of some 
quantitative trait on individuals, decomposed as 

Y = u + B + E 

where p is an unknown scalar parameter, where B and E are independent multivari- 
ate normal, Af(0,a 2 A) and J\f(Q,T 2 I), respectively, where a 2 and r 2 are unknown 
parameters called the additive genetic and environmental variance, respectively, A 
is a known matrix called the numerator relationship matrix in the animal breeding 
literature (Henderson, 1976), and I is the identity matrix. The matrix A is deter- 
mined solely by the relatedness of the individuals (relative to a known pedigree). 
Every element of A is nonzero if every individual is related to every other, and this 
implies all components of Y are correlated. 

In modern terminology, this is a mixed model with fixed effect vector fi and 
variance components a 2 and r 2 , but if the pedigree is haphazard so A has no regular 
structure, there is no scope for telling "n goes to infinity" stories like Miller (1977) 
does for mixed models for simple designed experiments. Our "no n" asymptotics 
do apply. The log likelihood may be nearly quadratic, in which case we have the 
"usual" asymptotics. 

The supplementary web site http://purl.umn.edu/92198 at the University of 
Minnesota Digital Conservancy gives details of two examples with 500 and 2000 
individuals. An interesting aspect of our approach is that its intimate connection 
with Newton's method (Section 4.5 and Appendix B) forced us to find a good 
starting point for Newton's method (a technology transfer from spatial statistics) 
and investigate its quality. Thus our theory helped improve methodology. 

Readers wanting extensive detail must visit the web site. The short summary is 
that the example with 500 individuals is not quite in asymptopia, the parametric 
bootstrap being needed for bias correction in constructing a confidence interval 
for logit heritability (loger 2 — logr 2 ). But when this example was redone with 
2000 individuals, the bias problem went away and the bootstrap could not improve 
asymptotics. 

2.6. Our Regularity Conditions 

Famously, Le Cam, although spending much effort on likelihood, did not like max- 
imum likelihood. Le Cam (1990) gives many examples of the failure of maximum 
likelihood. Some are genuine examples of bad behavior of the MLE. Others can 
be seen as problems with the "n goes to infinity" story as much as with maxi- 
mum likelihood. I have always thought that article failed to mention Le Cam's 
main reason for dislike of maximum likelihood: his enthusiasm for weakest possible 
regularity conditions. He preferred conditions so weak that nothing can be proved 
about the MLE and other estimators must be used instead (Le Cam and Yang, 
2000, Section 6.3). 

His approach does allow the "usual asymptotics of maximum likelihood" to be 
carried over to quite pathological models (Le Cam and Yang, 2000, Example 7.1) 
but only by replacing the MLE with a different estimator. The problem with this 
approach (as I see it) is that the resulting theory no longer describes the MLE, 
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hence is no longer useful to applied statisticians. (Of course, it would be useful if 
applied statisticians used such pathological models and such estimators. As far as 
I know, they don't.) 

Thus we stick with old-fashioned regularity conditions involving derivatives of 
the log likelihood that go back to Cramer (1946, Chapters 32 and 33). We shall 
investigate the consequences of being "nearly" LAMN in the sense that the log 
likelihood and its first two derivatives are near those of an LAMN model. These 
conditions arc about the weakest that still permit treatment of Newton's method, 
so our theorems apply to the way maximum likelihood is done in practice. Our 
approach has the additional benefit of making our theory no stranger than it has 
to be in the eyes of a typical statistician. 

3. Models with Quadratic Log Likelihood 

3.1. Log Likelihood 

The log likelihood for a parametric family of probability distributions having den- 
sities fg, 9 € O with respect to a measure A is a random function I defined by 

(4) 1(9) =u(X) + log f e (X), 9eO, 

where X is the random data for the problem and u is any real valued function on 
the sample space that does not depend on the parameter 9. In this article, we are 
only interested families of almost surely positive densities so the argument of the 
logarithm in (4) is never zero and the log likelihood is well defined. This means all 
the distributions in the family are absolutely continuous with respect to each other. 

Then for any bounded random variable g(X) and any parameter values 9 and 
9 + 5 we can write 

E g+5 {g(X)} = J g(x)f e+s (x)X(dx) 

= [ g(x)^^fe(x)X(dx) 

(5) J fe(x) 

= Jg(x)e l ^-Wf e (x)\(dx) 

= E e {g(X)e 1 ^- 1 ^} 

The assumption of almost surely positive densities is crucial. Without it, the second 
line might not make sense because of division by zero. 

3.2. Quadratic Log Likelihood 

Suppose the log likelihood is defined by (1). The random variables U, Z, and K 
are, of course, functions of the data for the model, although the notation does not 
indicate this explicitly. 

The constant term U in (1) analogous to the term u(X) in (4) is of no importance. 
We are mainly interested in log likelihood ratios 

(6) 1(9 + 8) - 1(9) = (Z- K6)'S - \8'K5, 
in which U does not appear. 
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Theorem 3.1 (LAMN). Suppose (1) is the log likelihood of a probability model, 
then 

(a) K is almost surely positive semi-definite. 

Also, the following two conditions are equivalent (each implies the other). 

(b) The conditional distribution of Z given K for parameter value 9 is N(KQ, K) . 

(c) The distribution of K does not depend on the parameter 9. 

Any model satisfying the conditions of the theorem is said to be LAMN (locally 
asymptotically mixed normal). Strictly speaking, "locally asymptotically" refers to 
sequences converging to such a model, such as those discussed in Section 4, but 
"MN" by itself is too short to make a good acronym and LAMN is standard in the 
literature. 

Our theorem is much simpler than traditional LAMN theorems (Le Cam and Yang, 
2000, Lemmas 6.1 and 6.3) because ours is not asymptotic and the traditional ones 
are. But the ideas are the same. 

Proof. The special case of (5) where g is identically equal to one with (6) plugged 
in gives 

(7) 1 = £0+5(1) = E B {e K > 2 ) 
Averaging (7) and (7) with 5 replaced by —8 gives 

(8) £ {cosh[(Z - K6)'5]ex-p[-±5'KS]} = 1. 

Now plug in sS for 8 in (8), where s is scalar, and use the fact that the hyperbolic 
cosine is always greater than one giving 

1 = E 9 {cosh[s{Z - ROYS] cxp[-±s 2 5'K5} } 
>Eg{ C M-\s 2 5'K5]I { ^_ t) {5'K5)} 
> exp(±s 2 e)P {5'K5 < -e} 

For any e > 0, the first term on the right hand side goes to infinity as s — > 00. 
Hence the second term on the right hand side must be zero. Thus 

P e {5'K5 < -e} = 0, e > 

and continuity of probability implies the equality holds for e — as well. This 
proves (a). 

Replace g(X) in (5) by h(K) where h is any bounded measurable function giving 

(9) E e+s {h(K)} = E 9 {h(K)e( z - Ke y 5 -5 s ' K5 }. 

Assume (b) . Then the moment generating function of Z given K for the param- 
eter 6 is 

(10) E 8 {e z ' s I K) = e e ' KS +\ s ' KS 
and this implies 

(11) E e {e( z - Ke y s -5 S ' K6 I K) = 1. 
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Plugging (11) into (9) and using the iterated expectation theorem we get 
E g+s {h(K)} = E {h(K)Ee{e^- Keys ~^' KS | A'}} 

= Eg{h(K)} 

which, h being arbitrary, implies (c). This proves (b) implies (c). 

Now drop the assumption of (b) and assume (c), which implies the left hand side 
of (9) does not depend on S, hence 

(12) Eg{h(K)} = E e {h(K)e (z - Keys -^ 5 ' KS }. 

By the definition of conditional expectation (12) holding for all bounded measurable 
functions h implies (11) and hence (10), which implies (b). This proves (c) implies 
(b). □ 

Corollary 3.2 (LAN). Suppose (1) is the log likelihood of an identifiable probability 
model and K is constant, then 

(a) K is positive definite. 
Moreover, 

(b) The distribution of Z for the parameter value 9 is J\f(K9, K). 

Any model satisfying the conditions of the corollary is said to be LAN (locally 
asymptotically normal). Strictly speaking, as we said about LAMN, the "locally 
asymptotically" refers to sequences converging to such a model, but we also use it 
for the model itself. 

Proof. The theorem applied to the case of constant K gives (a) and (b) with "pos- 
itive definite" in (a) replaced by "positive semi-definite." So wc only need to prove 
that identifiability implies positive definitencss. 

If K were not positive definite, there would be a nonzero 6 such that 5'KS = 0, 
but this would imply Kb = and the distribution of Z for the parameter value 
6 + 6 would be Af(K9,K). Hence the model would not be identifiable (since Z 
is a sufficient statistic, if the distribution of Z is not identifiable, neither is the 
model) . □ 

We cannot prove the analogous property, K almost surely positive definite, for 
LAMN. So we will henceforth assume it. (That this doesn't follow from identifia- 
bility is more a defect in the notion of identifiability than in LAMN models.) 



3.3. Examples and Non-Examples 

The theorem provides many examples of LAMN. Let K have any distribution that 
is almost surely positive-definite- valued, a Wishart distribution, for example. Then 
let Z | K be N{K0,K). 

The corollary provides a more restricted range of examples of LAN. They are 
the multivariate normal location models with nonsingular variance matrix that does 
not depend on the parameter. 

A non-example is the AR(1) autoregressive model with known innovation vari- 
ance and unknown autoregressive parameter. Let Xq have any distribution not 
depending on the parameter 9, and recursively define 

(13) X n = 9X n -i + Z n 
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where the Zi are independent and identically Af(0, 1) distributed. The log likelihood 



is 



i=l 

which is clearly quadratic with Hessian — K n = — X)"=i -^i-v From (13) 

S e (X2|X o ) = 2 i? ( ,(X2_ 1 |X o ) + l 
and from this we can derive 

n- 1 



£ e (X„|X ) = 



1 - 6> 2 



l-6» 2 



_ 02(n-l) 

1 -6» 2 



^1, 



which is enough to show that the distribution of X n depends on 9 so this model is 
not LAMN (in the "no n" sense we are using the term in this article, although it 
is LAN in the limit as n — > oo for some values of 8). 



4. Likelihood Approximation 

It is plausible that a model that is "nearly" LAN has an MLE that is "nearly" nor- 
mally distributed and a model that is "nearly" LAMN has an MLE that is "nearly" 
conditionally normally distributed. In order to make these vague statements math- 
ematically precise, we need to define what we mean by "nearly" and explore its 
mathematical consequences. 



4-1. Convergence in Law in Polish Spaces 

Since log likelihoods are random functions, it makes sense to measure how close 
one is to another in the sense of convergence in law. In order to do that, we need 
a theory of convergence in law for function- valued random variables. We use the 
simplest such theory: convergence in law for random variables taking values in 
a Polish space (complete separable metric space). Billingsley (1999, first edition 
1968) has the theory we will need. Other sources are Fristedt and Gray (1997) and 
Shorack (2000). 

A sequence of random elements X n of a Polish space S converges in law to 
another random element X of S if E{f(X n )} E{f(X)}, for every bounded 
continuous function / : S — > R. This convergence is denoted 

X n — ► X 

and is also called weak convergence, a term from functional analysis, and (when the 
Polish space is R p ) convergence in distribution. 

The theory of convergence in law in Polish spaces is often considered an advanced 
topic, but our use of it here involves only 

• the mapping theorem (Billingsley, 1999, Theorem 2.7) 

• the Portmanteau theorem (Billingsley, 1999, Theorem 2.1) 

• Slutsky's theorem (Billingsley, 1999, Theorem 3.1) 

• Prohorov's theorem (Billingsley, 1999, Theorems 5.1 and 5.2) 

• the subsequence principle (Billingsley, 1999, Theorem 2.6) 
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which should be in the toolkit of every theoretical statistician. They are no more 
difficult to use on random elements of Polish spaces than on random vectors. The 
last three of these are only used in Appendix C where we compare our theory to 
conventional theory. Only the mapping and Portmanteau theorems are needed to 
understand the main text. 

4.2. The Polish Spaces C(W) and C 2 (W) 

Let W be an open subset of W and let C(W) denote the space of all continuous 
real-valued functions on W. The topology of uniform convergence on compact sets 
for C(W) defines /„ / if 

sup|/ n (x) — f(x)\ — > 0, for every compact subset B of W. 

xes 

The topology of continuous convergence for C(W) defines /„ — > f if 
fn(x n ) -> f(x), whenever x n ->• x. 

Kuratowski (1966, Section 20, VIII, Theorem 2) shows these two topologies for 
C{W) coincide. Bourbaki (1998, Ch. 10, Sec. 3.3, Corollary, part (b)) shows that 
C(W) is a Polish space. 

Let C 2 (W) denote the space of all twice continuously diffcrcntiablc functions 
W — > R with the topology of uniform convergence on compact sets (or continuous 
convergence) of the functions and their first and second partial derivatives. Then 
C 2 (W) is isomorphic to a subspace of C(W) 1+p+pxp , in fact a closed subspace 
because uniform convergence of derivatives implies sequences can be differentiated 
term by term. Hence C 2 (W) is a Polish space, (Fristedt and Gray, 1997, Proposi- 
tions 2 and 3 of Section 18.1). 

We will consider an almost surely twice continuously diffcrcntiable log likelihood 
whose parameter space is all of W to be a random clement of C 2 {W) and will 
restrict ourselves to such log likelihoods (for measurability issues, see Appendix A). 
We find a way to work around the assumption that the parameter space be all of 
W in Section 4.6. 

4-3. Sequences of Statistical Models 

In order to discuss convergence we use sequences of models, but merely as technical 
tools. The n indexing a sequence need have nothing to do with sample size, and 
models in the sequence need have nothing to do with each other except that they 
all have the same parameter space, which is MP. As we said in Section 2.2, wc 
could eliminate sequences from the discussion if we wanted to. The models have log 
likelihoods l n and true parameter values ip n . 

Merely for comparison with conventional theory (Appendix C) we also introduce 
a "rate" t„. In conventional asymptotic theory r„ = yjn plays an important role, 
so we put it in our treatment too. In the "no n" view, however, where the models 
in the sequence have "nothing to do with each other" there is no role for r n to play, 
and we set r„ = 1 for all n. 

Define random functions q n by 

(14) q n (8) = Z„(Vn + t-H) - l n (tp n ), 5 e R p . 
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These are also log likelihoods, but we have changed the parameter from 9 to 5, so 
the true value of the parameter S is zero, and subtracted a term not containing 6, 
so q n (0) = 0. 

Our key assumption is 

(15) ?„A ? , inC 2 (R p ) 
where q is the log likelihood of an LAMN model 

(16) q(6) = S'Z - ±6'K5, 5 g R p , 

there being no constant term in (16) because q n (0) — for all n implies q(0) = 0. A 
consequence of LAMN is that e q ^ is for each S a probability density with respect 
to the measure governing the law of q, which is the measure in the LAMN model 
for 6 = because e q ^ = 1. This implies 

(17) E{e q ^} = l, SeR p . 

Please note that, despite our assuming LAMN with "N" standing for normal, 
we can say we are not actually assuming asymptotic normality. Asymptotic (con- 
ditional) normality comes from the equivalence of the two conditions in the LAMN 
theorem (Theorem 3.1). We can say that we are assuming condition (c) of the the- 
orem and getting condition (b) as a consequence. Normality arises here from the 
log likelihood being quadratic and its Hessian being invariant in law. Normality is 
not assumed, and the CLT plays no role. 

4-4- Contiguity 

In Le Cam's theory, property (17) is exceedingly important, what is referred to 
as contiguity of the sequence of probability measures having parameter values 
ip n + t~ 5 to the sequence having parameter values i/} n (Le Cam and Yang, 2000, 
Theorem 1 of Chapter 3). 

Contiguity (17) does not follow from the convergence in law (15) by itself. From 
that we can only conclude by Fatou's lemma for convergence in law (Billingslcy, 
1999, Theorem 3.4) 

(18) E{e q ^}<l, SeR p . 

That we have the equality (17) rather than the inequality (18) is the contiguity 
property. In our "no n" theory (17) arises naturally. We always have it because 
"improper" asymptotic models (having densities that don't integrate to one) make 
no sense. 

4-5. One Step of Newton's Method 

Our presentation is closely tied to Newton's method. This is a minor theme in 
conventional theory (one-step Newton updates of root-?i-consistent estimators are 
asymptotically equivalent to the MLE), but we make it a major theme, bringing 
our theory closer to actual applications. We consider two kinds of estimator: onc- 
step-Newton estimators, like those in conventional theory, and Newton-iterated- 
to-convergence estimators (Appendix B), what in applications are usually deemed 
MLE. 
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To treat these estimators we need to consider not only the log likelihood but also 
the starting point for Newton's method. The one-step Newton map G is defined for 
an objective function q and a current iterate 5 by 

(19) G(q,S) = 5+(-V 2 q(S))-\q(S) 

when — W 2 q(S) is positive definite (otherwise Newton's method makes no sense as 
an attempt at maximization). In order to deal with this possible failure of Newton's 
method, we allow S and G(q,5) to have the value NaO (not an object), which, like 
NaN (not a number) in computer arithmetic, propagates through all operations. 
Addition of a new isolated point to a Polish space produces another Polish space. 

Lemma 4.1. If q is strictly concave quadratic and 5 ^ NaO, then G(q,5) is the 
unique point where q achieves its maximum. 

Proof. Suppose 

(20) q(S) =u + z'S- \5'K5, 
where K is positive definite, so 

Vq{5) =z-K5 
V 2 q{8) = —K 

Then 

G(q,S) = 5 + R-\z-K5) = K~ 1 z, 

which is the solution to Vg(5) = 0, and hence by strict concavity, the unique global 
maximizer of q. □ 

Lemma 4.2. The Newton map G is continuous at points (g, S) such that q is strictly 
concave quadratic. 

Proof. Suppose 5 n — > S and q n — > q with q given by (20). If S = NaO the conclusion 
is trivial. Otherwise, because convergence in C 2 (M. P ) implies continuous convergence 
of first and second derivatives 

~V 2 q n (5 n ) -> -V 2 q(6) = K, 

which is positive definite, so G(q n ,S n ) 7^ NaO for sufficiently large n, and 

Vq n (S n ) -+Vq(6)=z- KS, 

so 

G(q n ,8 n ) -+G(q,5) =K- 1 z. 

□ 

Theorem 4.3. Let 8 n be a sequence of random elements ofM. d and q n a sequence 
of log likelihoods having parameter space M. d and true parameter value zero. Suppose 

(21) (q n ,6 n ) ^ M) 
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where q is the log likelihood of an LAMN model (16), and define 6„ = G(q n ,d n ) 
where G is given by (19). Then 



(22a) -VV(^) A K 

(22b) -V 2 q n {8 n ) A K 

(22c) S n -A K~ 1 Z 

(22d) (-V 2 g n (5„)) 1/2 5 n AaT(0,/) 



where the matrix square root in (22d) is i/ie symmetric square root when the matrix 
is positive definite and NaO otherwise. Moreover, (21), (22a), (22b), (22c), and 
(22d) /ioZd jointly. 

The convergence in law (21) takes place in C 2 (R P ) x R p , the product of Polish 
spaces being a Polish space (Billingslcy, 1999, Appendix M6). In (22d) / is the pxp 
identity matrix. Since the random variables on the right hand sides of all these limits 
are never NaO, the left hand sides are NaO with probability converging to zero. 

Some would rewrite (22d) as 

(23) 0„«AA(^„,(-V 2 /„(4)) _1 ) , 

where the double wiggle means "approximately distributed" or something of the 
sort and where we have shifted back to the original parameterization 

On = i'n + T~ X 8 n 

= G(l n ,8 n ) 

Strictly speaking, (23) is mathematical nonsense, having no mathematical content 
except by allusion to (22d), but it is similar to 

(24) n *M(l>,In0n)- 1 ) 

familiar from conventional treatments of the asymptotics of maximum likelihood, 
where ip is the true parameter value and I n (9) is expected Fisher information for 
sample size n. 

Proof. We claim the map 

q \ 

S 

V 2 q(5) 
V 2 q(G(q,S)) 
G(q,S) 

(-V 2 q(G(q,5))y /2 G(q,S)) 

is continuous on C 2 (M d ) x R 2 at points where q is strictly concave quadratic. By 
definition of product topology, (25) is continuous if each component is continuous. 
The first two components are also continuous by definition of product topology, the 
fifth by Lemmas 4.1 and 4.2, then the third and fourth by continuous convergence, 
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and the sixth by matrix multiplication being a continuous operation, and matrix 
square root being a continuous operation on the set of positive definite matrices 
(Horn and Johnson, 1985, Problem 7.2.18) and continuous convergence. 

Now all of the assertions of the theorem follow from the mapping theorem 
(Billingslcy, 1999, Theorem 2.7), the only non-obvious limit being (22d), which is 
clearly K~ 1 / 2 Z. Since Z is A/"(0, K) given K by q being LAMN with true parameter 
value zero, K-^ 2 Z is Af(0, 1) conditional on K. Since the conditional distribution 
does not depend on K, it is also the unconditional distribution. □ 

Corollary 4.4. If K = — V 2 q(<5) in the theorem is constant, then (22c) can be 
replaced by 

(26) 8 n -^M{Q,K- 1 ) 
The proof is obvious. Some would rewrite (26) as 

(27) 9 n « jV(t/>ni (r^Ky 1 ) , 

from which we see that K plays the role of expected Fisher information for sample 
size one and t 2 K plays the role of expected Fisher information for sample size n, 
though K isn't the expectation of anything in our setup. 

4-6. Bounding the Approximation 

Suppose we wish to use (22d) to produce a confidence region. Another application 
of the mapping theorem gives 

6' n {-V 2 q n (S n ))S n -A chi 2 (p), 

where the right hand side denotes a chi-square random variable with p degrees of 
freedom. If k is the upper a quantile of this distribution, then by the Portmanteau 
theorem (Billingslcy, 1999, Theorem 2.1) 

liminfPr{^(-VV(4))4 < 4 > 1 - a 

and, mapped to the original parameterization, this tells us that 

{ 9 e W : (§ n - 9)'(-W 2 l n (9 n )) n - 6) < k } 

is a 1 — a asymptotic confidence region for the true unknown parameter value ip n . 

So far so conventional, but now we want to do a finer analysis using the joint 
convergence of 9 n and 9 n . For any open set W we get 

(28) liminf Pr { S' n (-V 2 q n (5 n ))S n < n and S n £W and S n G w) 

> 1 - a - Pt(5 (£WorS(£W) 

(using Bonfcrroni on the right hand side), and now we note that we only evaluate 
Wq n and V 2 q n at S n and 5 n , and hence for this statement to hold we only need the 
convergence in law (21) relative to W, that is, for (28) to hold we only need (21) 
to hold in C 2 (W) x W. 

This section is our counterpart of the conventional "regularity condition" that 
the true parameter value be an interior point of the parameter space. But here we 
see that if 9 n is a very bad estimator of tp n , then W may need to be very large in 
order to have Pr(i5 ^ W orS £ W) very small. The conventional regularity condition 
vastly oversimplifies what is really needed. 
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5. Discussion 

It has not escaped our notice that the "no n" view advocated here leaves no place 
for a lot of established statistical theory: Edgeworth expansions, rates of conver- 
gence, the Bayes information criterion (BIC), and the subsampling bootstrap, to 
mention just a few. Can all of this useful theory be replaced by some "no n" analog? 
Only time will tell. Our goal is merely to explicate this "no n" view of maximum 
likelihood. We are not advocating political correctness in asymptotics. 

It does seem obvious that "no n" asymptotics based on LAN and LAMN theory 
says nothing about situations where no likelihood is specified (as in quasilikelihood, 
estimating equations, and much of nonparametrics and robustness) or where the 
likelihood is incorrectly specified or where the likelihood is correctly specified but 
the parameter is infinite-dimensional (as in nonparametric maximum likelihood). 
Thus it is unclear how this "no n" view can be extended to these areas. 

Another classical result that does not transfer to the "no n" worldview is the 
asymptotic efficiency of maximum likelihood. In an LAN model, it is true that 
the MLE is the best equivariant-in-law estimator (van der Vaart, 2000, Proposi- 
tion 8.4), but James-Stein estimators (James and Stein, 1961) show that the MLE 
is not the best estimator (where "best" means minimum mean squared error). The 
"no n" view has no room for other interpretations: if one likes James-Stein estima- 
tors, then one cannot also consider the MLE asymptotically efficient (because LAN 
models are already in asymptopia) . The argument leading to Le Cam's almost sure 
convolution theorem (van der Vaart, 2000, Section 8.6) cannot be transferred to 
the "no n" world (because it would have to prove the MLE as good as James-Stein 
estimators in LAN models, and it isn't). 

The application described in Section 2.4 convinced us of the usefulness of this "no 
n" view in spatial statistics, statistical genetics, and other areas where complicated 
dependence makes difficult the invention of reasonable "n goes to infinity" stories, 
much less the proving of anything about them. But having seen the usefulness of the 
"no n" view in any context, one wants to use it in every context. Having understood 
the power of "quadraticity" as an explanatory tool," many opportunities to use it 
arise. When a user asks whether n is "large enough" when the log likelihood is 
nowhere near quadratic, now the answer is "obviously not." When a user asks 
whether there is a need to reparametcrize when the Wald confidence regions go 
outside the parameter space, now the answer is "obviously." 

We imagine some readers wondering whether our ideas are mere "generalized 
abstract nonsense." Are we not essentially assuming what we are trying to prove? 
Where arc all the deltas and cpsilons and inequality bashing that one expects in 
"real" real analysis? We believe such "inequality bashing" should be kept separate 
from the main argument, because it needlessly restricts the scope of the theory. 
Le Cam thought the same, keeping separate the arguments of Chapters 6 and 7 in 
Le Cam and Yang (2000). 

For readers who want to see that kind of argument, we have provided Lemma C.l 
in Appendix C that says our "key assumption" (15) is weaker than the "usual 
regularity conditions" for maximum likelihood (Ferguson, 1996, Chapter 18) in all 
respects except the requirement that the parameter space be all of W, which we 
worked around in Section 4.6. A similar lemma using a different Polish space with 
weaker topology is Lemma 4.1 in Geyer (1994), where a "single convergence in 
law statement about the log likelihood" (the conclusion of the lemma) is shown 
to follow from complicated analytic regularity conditions (the hypotheses of the 
lemma), modeled after Pollard (1984, pp. 138 ff.). 
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Despite all the theory we present, our message is very simple: if the log likeli- 
hood is nearly quadratic with Hessian nearly equivariant in law, then the "usual" 
asymptotics of maximum likelihood hold. The only point of all the theory is to show 
that conventional theory, which does not support our message, can be replaced by 
theory that docs. 

Appendix A: Measurability in C(W) and C 2 (W) 

Let W be an open subset of R p , and let B n be an increasing sequence of compact 
subsets of W whose union is W. Then Rudin (1991, Example 1.44) gives an explicit 
metric for the space C(W) defined in section 4.2 



(29) d(f,g) 



max 



2-"||/-ff||s n 



->i i + II/-0||b„ 

where for any compact set B we define 

\\f\\ B = sup|/(a;)|. 

A map F : D, — > C(W) is measurable if and only if its uncurried form (uj, 9) i— > 
F(oj)(9) is Caratheodory, meaning 

• uj i y F(ui)(6) is measurable for each fixed 6 €W and 

• i — ^ F(lj)(6) is continuous for each fixed u E H 

(Rudin, 1991, Example 1.44; Aliprantis and Border, 1999, Corollary 4.23 and Theo- 
rem 4.54). The isomorphism between C 2 (W) and a closed subspace of C(W) 1+p+pxp 
means a function from a measurable space to C 2 (W) is measurable if and only if 
it and its first and second partial derivatives have Caratheodory uncurried forms. 

Appendix B: Newton Iterated to Convergence 

In this appendix we consider Newton's method iterated to convergence. Of course 
Newton need not converge, so we have to deal with that issue. Define Newton 
iterates 

So = 6 

S n = G(q,6 n -i), n > 0. 
If the sequence {5 n } converges, then we define 
(30) G°°{q,5) = lim S n 

n— } oo 

and otherwise we define G oc (q,5) = NaO. This function G°° is called the infinite- 
step Newton map. 

Lemma B.l. If q is strictly concave quadratic and 5 ^ NaO, then G°°(q,S) is the 
unique point where q achieves its maximum. 

Proof. By Lemma 4.1 the Newton sequence converges in one step in this case and 
has the asserted properties. □ 

Lemma B.2. The infinite-step Newton map G°° is continuous at points (q,S) such 
that q is strictly concave quadratic. 
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Proof. Suppose S n — > S and g„ — > q, and suppose q is strictly concave quadratic 
given by (20). Let S = K~ l z be the unique point at which q achieves its maximum, 
and let B be a compact convex set sufficiently large so that it contains 6 and 5 in 
its interior and 

sup q(6) < q(S), 

SedB 

where dB denotes the boundary of B. Let S be the unit sphere. Then 

(S,t)^t'(-V 2 q n (6)-K)t 

converges continuously to zero on B x S. Hence 

(31a) sup \t'(-V 2 q n (S) - K)t\ -> 0. 

SeB 
teS 

Also 

(31b) q n (S) - sup q n (6) -> q(6) - sup q(8) 

SedB SedB 

and 

(31c) q n (S n ) - sup q n (S) -> g(<5) - sup q(<5) 

SedB SedB 

and the limits in both (31b) and (31c) are positive. 

Choose N such that for all n > N the left hand side of (31a) is strictly less than 
one-third of the smallest eigenvalue of K and the left hand sides of both (31b) and 
(31c) are positive. For n > N, —\7 2 q n (S) is positive definite for 6 G B, hence q n is 
strictly concave on B. Let S n denote the (unique by strict concavity) point at which 
q n achieves its maximum over B. Since the left hand side of (31b) is positive, S n 
is in the interior of B and Vq n (8 n ) = 0. By Theorem 1 and Example 2 of Veselic 
(2004) Newton's method applied to q n starting at S n converges to S n for n > N. 

Consider the functions H n and H defined by H n (8) = G(q n ,S) and H(8) — 
G{q,5). Then for n> JV, we have 6 n = H n (S n ). By Lemma 4.2 H n converges to H 
uniformly on compact sets. By Lemma 4.1 S — H(S) whenever S ^ NaO. Thus, if 
W is a neighborhood of 8, then H n maps B into W for sufficiently large n, which 
implies S n G W for sufficiently large n, which implies 6 n — > 6. □ 

Theorem B.3. Theorem 4-3 holds with G replaced by G°° . 

Just change the proof to use Lemmas B.l and B.2 instead of Lemmas 4.1 and 4.2. 

When Newton's method converges, under these conditions (the Hessian is con- 
tinuous but not necessarily Lipschitz), then its convergence is superlincar (Fletcher, 
1987, Theorem 6.2.3). If the Hessian does happen to be Lipschitz, then Newton's 
method converges quadratically (Fletcher, 1987, Theorem 6.2.1). 

It is clear from the proof of Lemma B.2 that the argument of Section 4.6 carries 
over to this situation. Now we evaluate Vg„ and V 2 q n at an infinite sequence of 
points, but as the last paragraph of the proof makes clear, all but the first of 
these points lie in an arbitrarily small neighborhood of 5 n for sufficiently large n. 
There is "almost" no difference between one-step Newton and Newton iterated to 
convergence. 

Please note that this appendix is an endorsement of Newton's method only for 
objective functions that are "nearly" strictly concave quadratic so that Newton's 
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method "nearly" converges in one step. If Newton's method does not "nearly" 
converge in one step, then one is crazy to use it and should instead use some form 
of "safeguarding" such as line search (Fletcher, 1987, pp. 21-40) or trust regions 
(Fletcher, 1987, Chapter 5). 

Appendix C: Comparison with Classical Theorems 

This appendix compares our "regularity conditions" with the "usual regularity con- 
ditions" for maximum likelihood. So we leave "no n" territory and return to the 
conventional "n goes to infinity" story. We adopt "usual regularity conditions" sim- 
ilar to those of (Ferguson, 1996, Chapter 18). Suppose we have independent and 
identically distributed data Xi, X2, . . ., X n , so the log likelihood is the sum of 
independent and identically distributed terms 

n 

i=i 

where 

and where ip is the true parameter value. We assume 

(a) the parameter space is all of W, 

(b) hi is twice continuously diffcrcntiable, 

(c) There exists an integrable random variable M such that 

|| V 2 /ii(#)|| < M, for all 9 in some neighborhood of -0, 

(the norm here being the sup norm), 

(d) the expected Fisher information matrix 

K = -E^FPhity)} 

is positive definite, and 

(e) the identity J fg{x)\(dx) = 1, can be differentiated under the integral sign 
twice. 

Lemma C.l. Under assumptions (a) through (e) above, (15) holds with r n = y/ri. 
Proof. Assumption (e) implies 

var^V/i^)} = -E^{W 2 hi(ip)} 

by differentiation under the integral sign (Ferguson, 1996, p. 120), assumption (d) 
implies both sides arc equal to the positive definite matrix K, and the central limit 
theorem (CLT) and assumption (e) imply 

1 " 

(32) V ?B (0) = -= V V/iiW-) A Af(0, K). 

Define 

r n (S) = 6'Wq n (0)-^6'K6 
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and its derivatives 



Vr n (6) = Wq n (0)-KS 
V 2 r n (5) = -K 



We now show that 



(33a) r n A q, in C 2 (W) 

is a consequence of the mapping theorem (Billingsley, 1999, Theorem 2.7) and (32). 
Define a function F : W -> C 2 (W) by 

F(z)(S) = z'S-^S'KS 

We claim that F is continuous, which says no more than that 

z n — > z and 6 n —> S 

imply 

z'Jn - \5' n K5 n -> z'S - \S'K5 
z n — KS n — > z — KS 

which are obvious (and that second derivatives converge, which is trivial). Then an 
application of the mapping theorem in conjunction with (32) says 

(33b) F (Vg„(0)) A F(Y) 

where Y is a random vector that has the distribution on the right hand side of (32), 
and that is the desired conclusion: (33a) in different notation. 

Now we use Slutsky's theorem (Billingsley, 1999, Theorem 3.1). This says that 
if we can show 

9„-r„A0, mC 2 (W), 

then we are done. Using the isomorphism of C 2 (W) to a subspace of C(M. p ) 1+p+pxp 
and another application of Slutsky, it is enough to show the separate convergences 

(34a) q n -r n — > 

(34b) Vq n - Vr„ A 

(34c) V V - V 2 r n A 

which take place in C(E. P ), in C{R P ) P , and in C{W) pxp , respectively. But these are 
equivalent to 

p 

(35a) sup|g„(<5) - r n {S)\ — > 

5GB 

(35b) sup ||Vg n (<5) - Vr„(£)|| A 

seB 

(35c) sup \\V 2 q n (5) - V 2 r n (S)\\ A 

holding for every compact set B, which can be seen using the metric (29). 
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Let B(9, e) denote the closed ball in M. p centered at 9 with radius e. Then as- 
sumption (c) can be stated more precisely as the existence of an e > and an 
integrable random variable M such that 



(36) 
Define 



|V 2 /ii(#)|| < M, 0eB(il>,€). 



1 - 

H n (9) = — Y i V 2 h i (9) 



and 

H{9) = -E^{W 2 hi(6)}. 
Theorem 16(a) in Ferguson (1996) says that 



(37) 



sup \\H n (9)-H(6)\\ 

6£B(iP,e) 



and that H is continuous on B(ip,e), the latter assertion appearing in the proof 
rather than in the theorem statement. Note that H(ip) — K. Also note that 



Hence 



VV(<5) 
V 2 r„(<5) 



sup \\V 2 q n {5) - V 2 r„(<5)| 



sup 

5GB(0,7j) 



H n (9)-H(yj)\\ 



= sup 

< sup \\H n {9) -H(9)\\ 

ees(i/j,n- 1 /2 ?? ) 

+ sup \\H(0) - H(ip)\\ 

SGB^.n" 1 / 2 )?) 

and the first term on the right hand side is dominated by the left hand side of (37) 
for n such that n~ 1 / 2 ri < e and hence converges in probability to zero by (37), and 
the second term on the right hand side goes to zero by the continuity of H. Since 
this argument works for arbitrarily large i], it proves (35c). 

Now using some of the facts established above and the Maclaurin series 



Vq n (S) = V<z n (0) 



\7 2 q n (sS)S ds 



we get 



V<?„(£) - Vr n {6) 



i r 



H n (ip + n- 1/2 s5) -H(ip) 



8 ds. 



So 



sup \\Vq n (6)-Vr n (5)\\< sup sup H n {^ + n^sS) - H (ip) \\S\\ 

<5eB(0 : >j) 0<s<l 5gS(0,))) 



< sup 
SeB(o,ri) 



H n {i; + n- 1 / 2 d)~H{tl;) 



n 
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and we have already shown that the right hand side converges in probability to zero 
for any fixed 77, however large, and that proves (35b). 
Similarly using the Maclaurin series 



q n (S) = <5'V g „(0) + / S'V 2 q n (s5)5(l - s) ds 



we get 



q n (S) - r n (5) = / 6' \H n (iP + n^sS) - H{^)} 5(1 - s) ds 



and the argument proceeds similarly to the other two cases. 



□ 



Lemma C.2. The assumption (21) of Theorem 4-3 can be weakened to (15) and 
5 n being tight if the last assertion of the theorem is weakened by replacing (21) with 
(15). 

Proof. The random sequences S n and q n are marginally tight, either by assumption 
or by the converse half of Prohorov's theorem, hence jointly tight (Billingsley, 1999, 
Problem 5.9). Hence by the direct half of Prohorov's theorem there exist jointly 
convergent subsequences. For every subsequence (q nk ,5n k ) there is a convergent 
subsubscquence (21) with n replaced by n^. And this implies all the conclusions 
of the theorem with n replaced by . But since the limits are the same for all 
subsequences rife, it follows from the subsequence principle (Billingsley, 1999, The- 
orem 2.6) that the conclusions hold as is with n rather than nk r □ 

The point of Lemma C.2 is to justify Lemma C.l dealing only with q n instead 
of both q n and S n . In the conventional "n is sample size going to infinity" story, 
Lemma C.2 seems important, saying that it is enough that 9 n be a r n -consistcnt 
sequence of estimators. In the "no n" world, however, Lemma C.2 loses significance 
because T n -consistency is meaningless when n is not reified. 
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